Internet - Scale Data Management

I n the past few years, many areas of computer science have been abuzz with talk about “big data.” Although different people mean different things when they use this term, the consensus is that data volumes are exploding, and that new tools are needed to handle these volumes. EMC and IDC have released a series of white papers estimating that in 2009, there were 800 exabytes of digital data stored worldwide (1 exabyte = 1,000 petabytes = 1,000,000 terabytes; see www.emc.com/ digital_universe). By 2020, they predict that this number will be 44 times larger, or 35 zettabytes. (According to blogger Paul McNamara, if all of humanity were to post continuously to Twitter, it would take one century before the total volume of tweets occupied 1 zettabyte! See www.networkworld.com/community/ node/60897.) This flood of data is due in large part to the increased desire to collect, query, and mine logs, clickstreams, and other Internet-based data. This is increasingly true as companies look to monetize their data through targeted advertising, clickstream analysis, and customer behavior profiling. Such data isn’t just localized on a single machine but spread across many machines, which are often geographically distributed. This distribution arises as more and more datasets are available via the Internet, and as organizations themselves become larger and more geographically diverse. In addition, we’re realizing the need to give users more and exclusive control over their own data, as witnessed by the privacy discussions that surround online social networks. Requiring distributed control on top of geographical distribution further complicates data management. This demand for data processing has given rise to new technologies and techniques for Internet-scale data management. These include numerous “NoSQL” tools, designed to provide access to structured data without using a conventional relational database (often to address scalability or fault-tolerance concerns), as well as new relational systems aimed at operating at larger scales or higher rates than conventional databases. Additionally, researchers have developed several Sam Madden Massachusetts Institute of Technology